[SOLVED] [bash] 25,000 greps - better solution?

hashbang#! · 11-09-2010, 06:49 AM

colucix, sorry about the confusion, your result is spot-on - see #13.

I am not interested in the interim files. They were just a cludge for my 4-step solution. The only thing I need is appending the new ids to 2 files (as per my 1st post).

I need to sit down and work through your awk. I'd like to understand how it does it so I know better next time.

colucix · 11-09-2010, 07:19 AM

Quote:

Originally Posted by hashbang#!

colucix, sorry about the confusion, your result is spot-on - see #13.

Ok. Just seen the modification.

Quote:

Originally Posted by hashbang#!

I need to sit down and work through your awk. I'd like to understand how it does it so I know better next time.

Just take in mind that FILENAME is an internal variable storing the name of the file currently parsed. When you pass multiple filenames as argument, awk processes all of them in sequence and the FILENAME variable changes accordingly. For this reason, we have two rules in the code: the first one si executed for all the ids?????? file, the second one only for the idsmore file.

If something is still not clear, feel free to ask. FYI, my only and unique awk reference is the GNU official guide: http://www.gnu.org/software/gawk/manual/.

colucix · 11-09-2010, 07:34 AM

Quote:

Originally Posted by H_TeXMeX_H

Can you post the code to generate those, because I can't imagine in my head what needs to be done.

You can simply add more print statements at your pleasure. Since the code does not require sorting (as per the first lines of the OP's script), there is no need for the sorted temporary files... or were you really interested in how to perform sorting in awk?

Code:

FILENAME != "idsmore" {
  _[$0] = ""
}
FILENAME == "idsmore" {
  if (! ( $1 in _ )) {
    print $1 >> ( "ids" $2 $3 )
    print $1 >> ( "ids" $2 $3 ".log")
    print $1 >> "idsmissing"
    print >> "idsmissing_dated"
  }
}

H_TeXMeX_H · 11-09-2010, 07:45 AM

I was asking about the test files. I cannot write a solution without test files. You said you somehow made 5 files containing 56000 different ids each (total 280000) and a file "idsmore" containing 300000 different ids, how ? I don't really need that many. Or, if the problem is solved, just forget about it.

hashbang#! · 11-09-2010, 08:33 AM

colucix - got it. Ingenious!

colucix · 11-09-2010, 08:48 AM

@H_TeXMeX_H

Ops, sorry... I totally misunderstood your post. I generated 280000 numbers between 1 and 1000000 using the following awk code:

Code:

BEGIN {
  srand()
  do {
    num = 1 + int(rand() * 999999)
    if (! (num in _)) {
       print num
       _[num] = ""
       count++
    }
  } while ( count < 280000 )
}

then I cutted the output in five pieces using the split command and finally I added 20000 more numbers (plus dates) to generate idsmore. But for testing purposes you may also generate numbers in sequence between 1 and 280000 using seq.

H_TeXMeX_H · 11-09-2010, 08:49 AM

ok, thanks.

ntubski · 11-09-2010, 03:50 PM

Quote:

Originally Posted by hashbang#!

Am I understanding this correctly, that grep -F speeds up the operation because it does not try to interpret the patterns as regular expressions?

Yes, when grep knows that the patterns aren't regular expressions it can use the hash table approach, suggested by some posters here, internally.

Quote:

I get the same result without -F but it crawls. fgrep is a fraction of a second faster than grep -F.

You have to expect some random variation between different runs of the same program.

Quote:

egrep is the same as grep -E. fgrep is the same as grep -F. Direct invocation as either egrep or fgrep is deprecated, but is provided to allow historical applications that rely on them to run unmodified.

from http://www.manpagez.com/man/1/grep/

hashbang#! · 11-09-2010, 04:14 PM

ntubski, thanks for the info on grep. Changed my scripts accordingly.